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ABSTRACT 

The Effectiveness of Institutional Effectiveness: 

Doing an Institutional Effectiveness Survey on the Web 
The authors were interested in learning to what extent issues critical to the success of the institutional 
effectiveness process were addressed and in place at a variety of canpuses, and to what extent the institutional 
effectiveness process was deemed to be successful, useful, and valued by the campus communities. Secondly, the 
authors were interested in determining the effectiveness of e-mail and Web-based surveying as a mechanism for 
implementing a paperless communication process. Results suggest a widespread practice of institutional 
effectiveness among the nation's universities and colleges in ways that encourage broad participation in and 
ownership of the process among those campus communities. 
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The Effectiveness of Institutional Effectiveness: 

Doing an Institutional Effectiveness Survey on the Web 

With the demand for accountability increasing at the nation’s colleges and imiversities, institutional officers 
are growing more concerned about their ability to meet the high expectations of multiple constituencies. The 
growing need for limited resources puts increasing requirements on faculty, staff, and administrators to find ways to 
satisfy the increasingly vigilant constituents. College and university officials are finding students and parents 
demanding a quality education at affordable prices. At the same time, state and federal agencies are mandating 
college and university administrators, faculty, and staff to provide data which prove illustrate that they are doing 
what they say they are doing, as well as how well they are doing their job. 

Accreditation agencies such as the Southern Association of Colleges and Schools (SACS) require that all 
institutions in their accrediting region follow a general evaluation process that documents the evaluation of academic 
programs and administrative services (Criteria for Accreditation, 1996). This evaluation process is commonly 
referred to as institutional effectiveness. As a result, many college and university administrators and faculty are 
implementing institutional effectiveness processes that will assist their respective institutions to evaluate their 
internal and external effectiveness. However, the extent to which institutional effectiveness is successful and is seen 
as a useful addition to the campus operation varies from institution to institution. Several researchers have identified 
a number of indicators which point to the success or failure of institutional effectiveness on a caucus. Measurement 
of the institutional effectiveness efforts currently in practice, based on these indicators, can provide iir^ortant 
information for planners and practitioners. 

Of secondary interest to the researchers is reviewing the application of current Web technologies to survey 
research. The interest in paperless data-collection and reporting is a growing phenomenon among institutional 
research professionals. A number of carr^uses have developed and implemented Web-based data collection for 
their internal data needs, while some peer-group consortia share data with each other via Internet and Web-based 
communication, and many government and professional agencies outside the higher education field are also 
soliciting campuses for their data via the World Wide Web. The next step for paperless information exchange is on 
a campus-to-campus level by implementing a traditional use-and-application survey via the Web. 
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Literature Review 

Institutional Effectiveness 

A number of institutional research and assessment professionals have examined and defined the concepts of 
outcomes assessment and institutional effectiveness over the past twenty years. The Council on Postsecondary 
Accreditation provided an early view of outcomes assessment which they define as "the quality of an educational 
process relates to (1) the appropriateness of its objectives, (2) the effectiveness of the use of resources in pursuing 
these objectives and (3) the degree to which objectives is achieved. Without a clear statement of what education is 
expected to provide, it is not possible to determine how good it is" (1986). During the 1970s and 80s the concept of 
outcomes assessment was associated primarily with academic measures, and some educators saw a need for a 
concept more encompassing of the comprehensive range of institutional operations. 

The Commission on Colleges of the Southern Association of Colleges and Schools selected the term 
institutional effectiveness in 1985 as a broader, more representative descriptor of assessment activities beyond 
traditional academic departments. Grossman and Duncan (1989) stated that institutional effectiveness is the process 
of articulating the mission of the college, setting goals, defining how the college and the commxmity will know when 
these goals are being met, and using the data from assessment in an ongoing cycle of planning and evaluation. 

Alfred and Kreider (1991), however, identified it as the ability of an institution to produce the outcomes it desires in 
the publics it serves. In 1992 SACS stated that institutional effectiveness "involves a systematic, explicit, and 
documented comparison of institutional performance to institutional purpose" (Resource Manual on Institutional 
Effectiveness, p. 2). Recent authors put it in more practical terms. Roueche, Johnson, and Roueche (1997) define 
institutional effectiveness as “an internal strategy for planning and evaluating the generated data by which the 
college can determine if it is matching its performance to its purpose or simply as any activity (ies) by which any 
several key indicators were measured routinely” (p. viii). Roueche, et al.'s definition of institutional effectiveness 
clarifies the role and purpose of effectiveness on college and university campuses, and points to outcomes strategies 
as a method to improve the present and future course of institutions of higher education. 

Nichols (1995) and Roueche, et al. (1997) have identified a number of issues critical to the success of the 
institutional effectiveness process on a campus. A successful institutional effectiveness program incorporates many 
of the following principles and mechanisms: a presidential blessing of and active leadership for the process; broad 
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development utilizing many constituents across the cairq)us; adequate resources, including time, staff and 
equipment; faculty, staff, and administrative buy-in and support; leadership and support from the board of trustees; 
the campus understanding the process, importance and utility of institutional effectiveness; passion for the process; 
an understanding and recognition of institutional culture, context, and history; the ability to act on results; internal 
application of results beyond meeting accreditation requirements; and the ability and willingness to face and 
encounter change, the ability to utilize existing systems. 

The World Wide Web 

Part of the reason for the increased focus on surveying via the Web is due to the many advantages this 
medium offers. A frequently cited advantage is the costs associated with on-line as coirq)ared to traditional paper- 
and-pencil research. Schmidt (1997) found surveying on-line having benefits including "savings in both time and 
money for survey researchers" (p. 274). He also reported the reduction or elimination for paper resources and 
elimination of data entry expense and time (p. 275). Gjestland (1996) noted "because all interviews on the Internet 
are self-administered, there is a tremendous cost savings in not having interviewers" (p. 3). Gaddis reported (1998) 
that the "relatively low cost of designing and conducting an on-line survey" (p. 67) increases its appeal to 
researchers. 

Other advantages of on-line surveying have been reported. One such benefit is that researchers are not 
bound by geography (Gjestland, 1996; Smith & Leigh, 1997; Swoboda, Miihlberger, Weitkunat, Schneeweiss, 

1997). Smith and Leigh praised the accessibility of the Internet, as well as its potential to transmit a variety of 
communications including text, images, and sound (p. 496). Davis (1997) agreed with the conveniences or ease of 
access and added the benefit of "confidentiality of responses" (p. 2). 

Another reported benefit has been the ability to archive data resulting from on-line surveys (Smith & Leigh, 
1997; Schmidt, 1997). Several studies have applauded the rapid return of instruments when published on-line, 
which can lead to more rapid analysis (Gaddis, 1998; Gjestland, 1996; Swoboda et al., 1997). Gaddis indicated that, 
when relying on the Web for surveying, one is also less likely to have "interference from people or institutions that 
may prevent a survey from being delivered to a respondent" (p.67). 

Davis (1997) summed up these points by stating that "Internet surveys are fast, effective, and cost-efficient" 

(p. 1). 
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Apprehensions have been voiced that surveys using the Internet invite potentially biased results. Several 
researchers reported early findings that the majority of Internet users are young, white, educated, males (Gjestland, 
1996; Swoboda et al., 1997). In contrast, recent studies have reported that Internet users are becoming more 
representative of the population as a whole (Georgia Technical Institute’s Graphics, Visualization & Usability 
Center [GVU], 1999; Kaye & Johnson, 1999). However, the Graphics, Visualization and Usability Center's tenth 
World Wide Web User Survey reported a decrease in the number of female respondents. 

Other researchers have addressed concerns of the representativeness of participants from an e-mail survey. 
Smith and Leigh (1997) reported that analysis of responses from male participants did not show any significant 
differences across the method of the survey taken. Similarly, whether from the e-mail group or the paper-and-pencil 
group, female responses did not differ. They concluded, "obtaining similar patterns of responses, despite these 
differences in sample population, subject selection, siirvey administration, and testing environments, strongly argues 
in favor of the generalizability and validity of data collected from Internet subjects and hence the utility of using the 
Internet as an alternative or supplemental source of subjects" (p. 502). Heflich and Rice (1999) also found their e- 
mail survey to be "a viable method for obtaining reflective data from participants." Handwerk, Carson and 
Blackwell (2000) determined "that the demographic and attitudinal profiles of the on-line respondents were not 
significantly different from the paper-and-pencil respondents" (p. 13). 

Purpose and Rationale 

The authors were interested in learning to what extent a formalized institutional effectiveness process is in 
place at a variety of canpuses, and to what extent this process was deemed to be successful, useful, and valued by 
the campus communities. Secondly, the authors were interested in determining the effectiveness of Web-based 
surveying as a mechanism for implementing a paperless communication process. 

Method 

Sample 

The population of interest for this study is all college and university administrators in the United States who 
are responsible for or participate in the institutional effectiveness plan/process on their respective canpuses dining 
1999. Since identifying those individuals directly was not practical or realistic, the sampling frame consisted of all 
participants of at least one of the regional or national e-mail listservs or electronic newsletters serving 
college/university institutional research professionals dining the months of August and September 1999. Because 
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individual membership lists were not available to the researchers, and cross-membership in multiple listservs is 
possible, the sanple size can only be estimated at about 2,000 persons over the six-week term of the survey. 

Measure 

The researchers constructed a 2 3 -item survey designed to assess the respondents' opinions with regard to 
implementation of institutional effectiveness plans, programs and procedures on their campuses. The first four 
questions consisted of basic descriptors of the institution: student headcount ranges, type of institutional control 
(public or private) highest degree offered, and regional accrediting body. Remaining questions asked respondents to 
assess the existing institutional effectiveness plan on their campuses. Areas assessed included: perceived support 
for/leadership of the institutional effectiveness plan from the chief executive officer, board of trustees, faculty, and 
staffradministrators; the method of development of the institutional effectiveness plan; dispersal of responsibility for 
the institutional effectiveness effort; areas of the campus included in the institutional effectiveness effort; the 
institutional effectiveness cycle; the next regional accrediting body visit; integration of the institutional effectiveness 
plan into the caucus process; communication of results to the general campus; and integration of outcomes into 
campus planning. The survey concluded with an open-ended comment section. 

Design 

The researchers used a post-test only non-experiment design for this study. This design can be illustrated 
with the following notation: 

X O 

Where: 

X = the current state of institutional effectiveness on college/university campuses 

O = the measure or observation 

This type of design is weak with respect to internal validity and causal assessment, and is best applied to 
collection of information for describing the current state of a population with regard to the question of concern. 
Procedure 

The researchers visited the Association Institutional Research (AIR) Website (http://www.airweb.org) to 
identify listservs or electronic newsletters for the national and each of the regional professional institutional research 
organizations. In early August 1999, the initial e-mail soliciting participation in the survey was sent to each listserv. 
Three reminder e-mails were sent out at approximate two-week intervals over the following six-week period. Each 
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e-mail described the nature of the research and directed interested participants to the Web address where the survey 
was located. Due to the nature of listserv administration, it was necessary for the lead correspondent to join several 
of the listservs to have permission to submit the e-mails. In one instance, a listserv administrator offered to act as an 
intermediary who received the e-mails at his personal e-mail address, then submitted them to his listserv on behalf of 
the researchers. At the conclusion of the six-week collection window, the Web address to the survey was re-directed 
to a “survey closed” Web page. 

Results 

There were 324 participants in the final sairq)le for this study. Persons working for institutions whose 
highest degree granted is the associate's degree accounted for 114(112 from public institutions and 2 from private). 
Those whose institutions grant bachelor's degrees numbered 24 (3 public, 21 private), and those whose institutions 
grant graduate degrees accounted for 185 (107 public, 78 private) of the participants. 

Participants ranked their institution size, as defined by Fall term student headcount, among one of five size 
distributions. Eighteen were from institutions of 1,000 and under, 128 with 1,001 to 5,000, 64 with 5,001 to 10,000, 
37 from 10,001 to 15,000, and 77 from institutions with 15,000 or more students. 

Participants also indicated the regional accrediting body of their institution. The largest number of 
participants was in the Southern Association (41%), followed by 23% from North Central Association, 15% from 
Middle States Association, 10% Western, 6% Northwest, and 4% from institutions regulated by the New England 
Association. 

Of all participants, 67% indicated their institutions do have an institutional effectiveness plan in place, 23% 
do not, and 1 1% are not certain. Not all institutions have established institutional effectiveness plans. Significant 
differences (p < .01) were foimd across a couple of descriptive variables. Respondents from public institutions were 
more likely (x^ (2, N = 324) = 12.43) than those from private institutions to reply that they have a plan in place (see 
Table 1). Differences also arose across highest degree granted by the institution (4, N = 324) = 24.75), with 
respondents from schools granting associate degrees (82%) far outpacing those from both graduate degree (59%) 
and bachelor degree (52%) granting institutions in terms of already having a plan established (See Table 2). 

When asked to rate the perceived support of their canpus institutional effectiveness plan, 87% of 
respondents stated their chief executive officer supports it, 73% indicated their staff and administration support it, 
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70% reported their Board of Trustees supports it, and 50% perceived their faculty supports the campus institutional 
effectiveness plan. 



Table 1 . Institution control by campus having an institutional effectiveness plan or program in place 



Institution control 


Row 

Count 


Has IE 
plan 


Does not 
have IE 
plan 


Not 

certain 


Public 


223 


73% 


18% 


9% 


Private 


99 


53% 


32% 


15% 


TOTAL 


322 


67% 


23% 


11% 



12.4(df,N= )p<0.01 



Table 2. Highest degree granted by campus has an institutional effectiveness (IE) plan or program in place 



Highest degree granted 


Row 

Count 


Has IE 
plan 


Does not 
have IE 
plan 


Not 

Certain 


Associate is highest degree granted 


114 


82% 


16% 


3% 


Bachelors is highest degree granted 


25 


52% 


40% 


8% 


Graduate degrees are granted 


183 


59% 


245% 


16% 


TOTAL 


322 


67% 


23% 


11% 



Significant difference at p < 0.01 level, - 24.8 



Participants also rated the means by which the institutional effectiveness plan was developed on their 
campuses. Of those with a plan in place, 46% said it was developed in a collaborative process. The next highest 
type, 21%, was mostly as a top-down process, 16% reported their campuses used mostly a bottom-up process, 6% 
said it was primarily a bottom-up process, and 5% replied their campuses employed a primarily top-down mandate 
to develop their institutional effectiveness plan. 

Primary responsibility for implementation of the institutional effectiveness plan was identified as either by 
institutional committee (41%), one staff or administrator, (15%), administrative committee with no faculty 
representation (6%), one faculty member (1%), or academic committee (less than 1%). Write-in responses include 
academic departments, associate/assistant deans, and president’s council. 

Respondents rated four general categories of can^us areas included in the institutional effectiveness plan. 
Academic departments are most likely to be included (77%), while administration is included in the evaluation 
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process of 64% of respondents, support staff by 57%, and operations/maintenance are included by 56% of 
respondents. 

Respondents rated their campus for general attitude toward institutional effectiveness. The majority 
reported their campuses having mixed (35%), or welcoming/positive (32%) attitudes toward institutional 
effectiveness. Those perceiving a cynical/hostile attitude on their campuses accoimted for 17% of respondents, and 
15% stated there is a neutral attitude with regard to institutional effectiveness on their campuses. 

Campus attitude toward institutional effectiveness and perceived total support for it was significantly 
related (p < .001). Campuses rated for broader support were more likely (8, N = 97) = 50.01) to also be rated 
having a welcoming or positive attitude toward the institutional effectiveness process (See Figure 1). 



70% 

60% 

50% 

40% 

30% 

20 % 

10 % 

0% 



^Significant difference (p<0.01) occurred across 
campus attitude toward plan. 



@ No Support U Support from One 

□ Support from Two Q Support from Three 
M Support from Four 




Figure 1. Campus attitude toward institutional effectiveness plan by support of President/Chancellor, Board 
of Trustees, faculty, and staff/administration. 

Respondents rated how integral the institutional effectiveness plan is to the campus process. The majority 
(40%) saw it as a mix between being a tool for internal iii^rovement and an administrative task. Those rating the 
plan mostly as a tool for internal improvement numbered 25%, and those rating it mostly as an administrative task 
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comprised 16%. Extreme ends of the scale showed 10% viewing it primarily as a tool for internal iiiqjrovement, 
followed closely by 9% rating it primarily as an administrative task. 

Significant differences were found for perceived support of the institutional effectiveness plan and degree 
to which the plan was integral to the campus process (p < .001). Respondents who rate their campuses with broader 
support were more likely (x^ (8, N= 249) = 62.78) to rate the institutional effectiveness plan as being mostly or 
primarily a tool for internal improvement (See Figure 2). 



70% 

60% 

50% 

40% 

30% 

20% 

10% 

0% 



* Significant difference (p<0.01) occurred 
across how institutional effectiveness plan is 

Figure 2. How integral the institutional effectiveness plan is to the campus process by support of 

President/Chancellor, Board of Trustees, faculty, and staff/administration. 



H No S upport H Support from One 

□ Support from Two Q Support from Three 
■ Support from Four 




How institutional effectiveness plan is seen* 



The results from the institutional effectiveness process were communicated to the campus using a variety of 
means. The most common single method was in a newsletter published by the institutional effectiveness committee 
(14%), followed by Web page updates (1 1%) presidential memo (9%) and e-mail announcements (6%). Other 
methods used were reports, meetings, presentations, and combinations of the above. 

Respondents rated the level of integration of the institutional effectiveness plan into the campus process, 
with 61% believing it has some integration on their campuses, 29% believing it has low integration, and 10% 
believing it has high integration on their campuses. When cross-tabulated for level of support, significant 
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differences (p < .001) were noted. Campuses with broad support were more likely (8, N= 249) = 65.34) to have 
the institutional effectiveness plan highly integrated into their campus process (See Figure 3). 




How well outcomes from the IE process are integrated into the plaiming process* 



* Significant difference (p<0.01) occurred across how 
institutional effectiveness plan is seen. 



^No Support H Support from One 

□ Support from Two □ Support from Three 
■ Support from Four 



Figure 3. How well outcomes from the institutional effectiveness process are integrated into the planning 



process by support of President/Chancellor, Board of Trustees, faculty, and staff/administration. 



Discussion 

Two-thirds of respondents report an institutional effectiveness plan in place on their campuses, with nearly 
half the plans developed as a collaborative effort on the campuses. This finding suggests a widespread practice of 
institutional effectiveness among the nation’s universities and colleges in ways that encourage broad participation in 
and ownership of the process among those campus communities. Such practice underscores the objective that 
institutional effectiveness be a long-term endeavor important and integral, rather than separate or unrelated, to the 
operations of the campus. 

Higher support for the institutional effectiveness plan goes hand-in-hand with it being viewed as important 
to the campus internally. In addition, broad support for the institutional effectiveness plan relates strongly with 
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more positive attitudes about it, and with its outcomes being integrated into planning efforts. These findings suggest 
the continued use and successful application of institutional effectiveness may hinge upon it being an organic 
development of within the context of the campus needs and operations. 

Academics and administration lead in areas likely to be evaluated. Review of support staff and 
operations/maintenance, however, are reported by over half of respondents. The results here are not entirely 
surprising in that assessment activities have historically been focused on academic areas. What is interesting is the 
growing percentage in other areas not known for these kinds of activities — namely in the support staff and 
operations/maintenance. The rather high percentage is indicative of the increasing demands for accountability and 
the continued mandates by accrediting agencies for institutional effectiveness, meaning assessment and evaluation 
processes across the spectrum of the institution. 

The implementation of institutional effectiveness plans is most often seen as a job requiring the attention of 
more than one individual. Once the plans are created, it is often up to a committee or group to implement them. 

The majority of respondents with institutional effectiveness plans indicate either that a formal institution- wide 
committee, a more narrowly arranged committee, or an informal grouping of professionals are most likely to be 
responsible for implementation of the plan. Very few respondents report that the responsibility for discharging of the 
plan is placed in the hands of one single individual. Perhaps the scope of the plans, which often evaluate academic 
areas, administration, staff and operations are too broad to be comfortably executed by a lone manager. 

It is also interesting to note that only a handful of respondents report that institutional research offices are 
charged with the responsibility of carrying out the plan at their institutions. The role of institutional research offices 
in the development and implementation of institutional effectiveness plans was not specifically addressed in the 
instrument and could be a focus of future research. However, since invitations to participate in this study were sent 
to subscribers of regional institutional research associations and the majority of titles supplied by participants 
contained "institutional research," it may safely be assumed that institutional research professionals are, at a 
minimum, aware of the institutional effectiveness plans on their canqjuses. 

The researchers encountered several technical issues surrounding the specific method of data collection 
utilizing listserv communication and the Web form. Following the initial contact, two respondents requested that 
they not receive the e-mail reminders and five had problems accessing the Web page to conq^lete the survey. One 
disadvantage to using listservs for communication is that researchers cannot control who does or does not receive 
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the communications. Such lack of control might induce ill will among participants. When possible, it would be best 
for the researcher to acquire direct lists of e-mail addresses so that individual communication and participation can 
be controlled. With regard to Web page access, it is important the Web-form designer be familiar with as many 
Web-browsers and computer configurations as possible and have contingency plans so the interested participant can 
con^lete the survey even if his or her browser is incompatible. Such contingencies can include directing the 
participant to try another Web-accessible con^uter, making a downloadable form available for the participant to 
return via file transfer protocol (FTP) or e-mail, or to send an e-mail version of the survey for the participant to 
con^lete. Positive outcomes of the Web-form collection included three respondents con^limenting the speed and 
ease-of-use of the survey, and relevant application of the technology. 
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